105 research outputs found
Hardware-efficient on-line learning through pipelined truncated-error backpropagation in binary-state networks
Artificial neural networks (ANNs) trained using backpropagation are powerful
learning architectures that have achieved state-of-the-art performance in
various benchmarks. Significant effort has been devoted to developing custom
silicon devices to accelerate inference in ANNs. Accelerating the training
phase, however, has attracted relatively little attention. In this paper, we
describe a hardware-efficient on-line learning technique for feedforward
multi-layer ANNs that is based on pipelined backpropagation. Learning is
performed in parallel with inference in the forward pass, removing the need for
an explicit backward pass and requiring no extra weight lookup. By using binary
state variables in the feedforward network and ternary errors in
truncated-error backpropagation, the need for any multiplications in the
forward and backward passes is removed, and memory requirements for the
pipelining are drastically reduced. Further reduction in addition operations
owing to the sparsity in the forward neural and backpropagating error signal
paths contributes to highly efficient hardware implementation. For
proof-of-concept validation, we demonstrate on-line learning of MNIST
handwritten digit classification on a Spartan 6 FPGA interfacing with an
external 1Gb DDR2 DRAM, that shows small degradation in test error performance
compared to an equivalently sized binary ANN trained off-line using standard
back-propagation and exact errors. Our results highlight an attractive synergy
between pipelined backpropagation and binary-state networks in substantially
reducing computation and memory requirements, making pipelined on-line learning
practical in deep networks.Comment: Now also consider 0/1 binary activations. Memory access statistics
reporte
Event-Driven Contrastive Divergence for Spiking Neuromorphic Systems
Restricted Boltzmann Machines (RBMs) and Deep Belief Networks have been
demonstrated to perform efficiently in a variety of applications, such as
dimensionality reduction, feature learning, and classification. Their
implementation on neuromorphic hardware platforms emulating large-scale
networks of spiking neurons can have significant advantages from the
perspectives of scalability, power dissipation and real-time interfacing with
the environment. However the traditional RBM architecture and the commonly used
training algorithm known as Contrastive Divergence (CD) are based on discrete
updates and exact arithmetics which do not directly map onto a dynamical neural
substrate. Here, we present an event-driven variation of CD to train a RBM
constructed with Integrate & Fire (I&F) neurons, that is constrained by the
limitations of existing and near future neuromorphic hardware platforms. Our
strategy is based on neural sampling, which allows us to synthesize a spiking
neural network that samples from a target Boltzmann distribution. The recurrent
activity of the network replaces the discrete steps of the CD algorithm, while
Spike Time Dependent Plasticity (STDP) carries out the weight updates in an
online, asynchronous fashion. We demonstrate our approach by training an RBM
composed of leaky I&F neurons with STDP synapses to learn a generative model of
the MNIST hand-written digit dataset, and by testing it in recognition,
generation and cue integration tasks. Our results contribute to a machine
learning-driven approach for synthesizing networks of spiking neurons capable
of carrying out practical, high-level functionality.Comment: (Under review
Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines
Recent studies have shown that synaptic unreliability is a robust and
sufficient mechanism for inducing the stochasticity observed in cortex. Here,
we introduce Synaptic Sampling Machines, a class of neural network models that
uses synaptic stochasticity as a means to Monte Carlo sampling and unsupervised
learning. Similar to the original formulation of Boltzmann machines, these
models can be viewed as a stochastic counterpart of Hopfield networks, but
where stochasticity is induced by a random mask over the connections. Synaptic
stochasticity plays the dual role of an efficient mechanism for sampling, and a
regularizer during learning akin to DropConnect. A local synaptic plasticity
rule implementing an event-driven form of contrastive divergence enables the
learning of generative models in an on-line fashion. Synaptic sampling machines
perform equally well using discrete-timed artificial units (as in Hopfield
networks) or continuous-timed leaky integrate & fire neurons. The learned
representations are remarkably sparse and robust to reductions in bit precision
and synapse pruning: removal of more than 75% of the weakest connections
followed by cursory re-learning causes a negligible performance loss on
benchmark classification tasks. The spiking neuron-based synaptic sampling
machines outperform existing spike-based unsupervised learners, while
potentially offering substantial advantages in terms of power and complexity,
and are thus promising models for on-line learning in brain-inspired hardware
Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware
In recent years the field of neuromorphic low-power systems that consume
orders of magnitude less power gained significant momentum. However, their
wider use is still hindered by the lack of algorithms that can harness the
strengths of such architectures. While neuromorphic adaptations of
representation learning algorithms are now emerging, efficient processing of
temporal sequences or variable length-inputs remain difficult. Recurrent neural
networks (RNN) are widely used in machine learning to solve a variety of
sequence learning tasks. In this work we present a train-and-constrain
methodology that enables the mapping of machine learned (Elman) RNNs on a
substrate of spiking neurons, while being compatible with the capabilities of
current and near-future neuromorphic systems. This "train-and-constrain" method
consists of first training RNNs using backpropagation through time, then
discretizing the weights and finally converting them to spiking RNNs by
matching the responses of artificial neurons with those of the spiking neurons.
We demonstrate our approach by mapping a natural language processing task
(question classification), where we demonstrate the entire mapping process of
the recurrent layer of the network on IBM's Neurosynaptic System "TrueNorth", a
spike-based digital neuromorphic hardware architecture. TrueNorth imposes
specific constraints on connectivity, neural and synaptic parameters. To
satisfy these constraints, it was necessary to discretize the synaptic weights
and neural activities to 16 levels, and to limit fan-in to 64 inputs. We find
that short synaptic delays are sufficient to implement the dynamical (temporal)
aspect of the RNN in the question classification task. The hardware-constrained
model achieved 74% accuracy in question classification while using less than
0.025% of the cores on one TrueNorth chip, resulting in an estimated power
consumption of ~17 uW
Forward Table-Based Presynaptic Event-Triggered Spike-Timing-Dependent Plasticity
Spike-timing-dependent plasticity (STDP) incurs both causal and acausal
synaptic weight updates, for negative and positive time differences between
pre-synaptic and post-synaptic spike events. For realizing such updates in
neuromorphic hardware, current implementations either require forward and
reverse lookup access to the synaptic connectivity table, or rely on
memory-intensive architectures such as crossbar arrays. We present a novel
method for realizing both causal and acausal weight updates using only forward
lookup access of the synaptic connectivity table, permitting memory-efficient
implementation. A simplified implementation in FPGA, using a single timer
variable for each neuron, closely approximates exact STDP cumulative weight
updates for neuron refractory periods greater than 10 ms, and reduces to exact
STDP for refractory periods greater than the STDP time window. Compared to
conventional crossbar implementation, the forward table-based implementation
leads to substantial memory savings for sparsely connected networks supporting
scalable neuromorphic systems with fully reconfigurable synaptic connectivity
and plasticity.Comment: Submitted to BioCAS 201
Gibbs Sampling with Low-Power Spiking Digital Neurons
Restricted Boltzmann Machines and Deep Belief Networks have been successfully
used in a wide variety of applications including image classification and
speech recognition. Inference and learning in these algorithms uses a Markov
Chain Monte Carlo procedure called Gibbs sampling. A sigmoidal function forms
the kernel of this sampler which can be realized from the firing statistics of
noisy integrate-and-fire neurons on a neuromorphic VLSI substrate. This paper
demonstrates such an implementation on an array of digital spiking neurons with
stochastic leak and threshold properties for inference tasks and presents some
key performance metrics for such a hardware-based sampler in both the
generative and discriminative contexts.Comment: Accepted at ISCAS 201
Clinical prognostic indicators of dysphagia following prolonged orotracheal intubation in ICU patients
Introduction The development of postextubation wallowing dysfunction is well documented in the literature with high prevalence in most studies. However, there are relatively few studies with specific outcomes that focus on the follow-up of these patients until hospital discharge. The purpose of our study was to determine prognostic indicators of dysphagia in ICU patients submitted to prolonged orotracheal intubation (OTI). Methods We conducted a retrospective, observational cohort study from 2010 to 2012 of all patients over 18 years of age admitted to a university hospital ICU who were submitted to prolonged OTI and subsequently received a bedside swallow evaluation (BSE) by a speech pathologist. The prognostic factors analyzed included dysphagia severity rate at the initial swallowing assessment and at hospital discharge, age, time to initiate oral feeding, amount of individual treatment, number of orotracheal intubations, intubation time and length of hospital stay. Results After we excluded patients with neurologic diseases, tracheostomy, esophageal dysphagia and those who were submitted to surgical procedures involving the head and neck, our study sample size was 148 patients. The logistic regression model was used to examine the relationships between independent variables. In the univariate analyses, we found that statistically significant prognostic indicators of dysphagia included dysphagia severity rate at the initial swallowing assessment, time to initiate oral feeding and amount of individual treatment. In the multivariate analysis, we found that dysphagia severity rate at the initial swallowing assessment remained associated with good treatment outcomes. Conclusions Studies of prognostic indicators in different populations with dysphagia can contribute to the design of more effective procedures when evaluating, treating, and monitoring individuals with this type of disorder. Additionally, this study stresses the importance of the initial assessment ratings
- …